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j/^  GAO 

^^^^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Acconnting  Office 
Washington,  DC  20548 


July  12,  2002 

The  Honorable  Christopher  Shays 
Chairman,  Subcommittee  on  National  Security, 

Veterans’  Affairs,  and  International  Relations 
Committee  on  Government  Reform 
House  of  Representatives 

Dear  Mr.  Chairman: 

In  1996,  the  Department  of  Defense,  through  the  Air  Force,  launched  an 
acquisition  program  to  develop  and  produce  a  revolutionary  laser  weapon 
system.  The  system,  known  as  the  Airborne  Laser  because  it  is  being 
developed  for  installation  in  a  modified  Boeing  747  aircraft,  is  intended  to 
destroy  enemy  ballistic  missiles  almost  immediately  after  their  launch  (in 
the  so-called  “boost  phase”)  before  they  pose  a  threat  to  civilian 
populations  and  military  assets.  The  Air  Force  originally  estimated 
development  costs  at  $2.6  billion  and  projected  fielding  of  the  system  to 
begin  in  2006.  However,  by  August  2001,  the  Air  Force  determined  that 
maturing  the  technologies  and  developing  the  system  would  cost  about 
50  percent  more  and  take  another  4  years.  The  development  cost  estimate 
increased  to  $3.7  billion,  and  the  fielding  date  slipped  to  2010. 

Against  this  backdrop  of  cost  increases  and  schedule  delays,  the 
Department  of  Defense  transferred  responsibility  for  the  Airborne  Laser  in 
October  2001  from  the  Air  Force  to  the  Ballistic  Missile  Defense 
Organization,  which  shared  responsibility  with  the  armed  services  for 
developing  ballistic  missile  defense  systems.  Subsequently,  in  January 
2002,  the  Secretary  of  Defense  announced  broad  changes  to  the 
department’s  strategy  for  developing  and  acquiring  missile  defense 
systems.  Specifically,  the  Defense  Secretary  designated  the  Ballistic 
Missile  Defense  Organization  as  the  Missile  Defense  Agency  and  granted 
the  agency  expanded  responsibility  and  authority.  The  Secretary  directed 
the  agency  to  develop  an  integrated  system  with  various  elements  that 
have  the  capability  to  attack  enemy  missiles  in  all  phases  of  their  flight, 
transition  responsibility  for  the  production  and  fielding  of  systems  to  the 
individual  services,  and  encourage  incremental  improvements  by  inserting 
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new  technologies  through  a  series  of  block  upgrades.^  The  Airborne  Laser 
is  one  of  many  systems  affected  by  the  new  strategy. 

Concerned  about  significant  cost  and  schedule  problems  associated  with 
developing  the  Airborne  Laser,  you  asked  us  to  determine  (1)  why  the 
system’s  development  is  costing  more  and  taking  longer  than  the  Air  Force 
originally  estimated  and  (2)  whether  the  Missile  Defense  Agency’s  new 
strategy  for  developing  the  Airborne  Laser  incorporates  the  practices  that 
characterize  successful  programs. 


Results  in  Brief 


The  Air  Force  was  unable  to  meet  the  Airborne  Laser’s  original  cost  and 
schedule  goals  because  it  established  those  goals  before  it  fully 
understood  the  level  of  effort  that  would  be  required  to  develop  the 
critical  system  technology  needed  to  meet  the  user’s  requirements.  When 
the  Air  Force  launched  the  Airborne  Laser  acquisition  program, 
Department  of  Defense  policy  required  that  program  cost  and  schedule 
goals  be  established.^  In  1996,  at  program  launch,  the  Air  Force  did  not 
have  enough  knowledge  about  the  technology  challenges  facing  the 
program.  As  a  result,  the  Air  Force  underestimated  the  complexity  of  the 
engineering  task  at  hand  and  misjudged  the  amount  of  time  and  money 
that  the  program  would  need.  Some  critical  technologies  that  the  system’s 
design  depends  upon  remain  immature,  making  it  very  difficult,  even 
today,  for  analysts  to  establish  realistic  cost  and  schedule  goals. 

The  Missile  Defense  Agency’s  new  strategy  for  developing  the  Airborne 
Laser  incorporates  some  knowledge-based  practices  that  characterize 
successful  programs.  For  example,  one  practice  that  the  agency 
implemented  is  a  requirements  process  that  gives  the  agency  the  flexibility 
to  develop  a  system  that  has  some  capability  without  being  held  to 
requirements  that  cannot  be  met  with  currently  available  technology.  A 
second  knowledge-based  practice  is  the  provision  of  additional  time  and 
facilities  for  testing.  Increased  testing  allows  agency  officials  to  reach  a 
better  understanding  of  the  capabilities  of  the  technology  so  that  they  can 
establish  more  realistic  requirements  and  ultimately  more  accurate 


^  The  Missile  Defense  Agency  plans  to  develop  a  series  of  Airborne  Laser  configurations, 
which  are  referred  to  as  “blocks.”  It  is  expected  that  each  block  will  include  improved 
technology  that  was  not  available  in  the  prior  block. 

^  This  policy  implements  statutory  planning  and  reporting  requirements  for  major  defense 
acquisition  programs. 
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estimates  of  the  time  and  money  needed  to  meet  those  requirements.  A 
third  practice  is  the  collection  of  the  types  of  information  needed  to 
determine  whether  the  technology  is  “in-hand”  to  give  war  fighters  an 
Airborne  Laser  with  some,  if  not  all,  desired  capabilities.  For  example,  the 
agency  intends  to  compare  developed  capabilities  with  data  derived  from 
intelligence  sources  on  the  likely  launch  points  and  types  of  missiles  that 
the  system  could  encounter. 

However,  the  agency  has  not  established  knowledge-based  decision  points 
and  associated  criteria  for  moving  forward  from  technology  development 
to  product  development  and  on  to  production.  Separating  technology 
development  from  product  development  has  been  a  critical  determinant 
for  successful  program  outcomes.  Without  decision  points  and  criteria,  the 
agency  risks  beginning  new  and  more  costly  activities  before  it  has  the 
knowledge  to  determine  the  money  and  time  required  to  complete  the 
activities  and  whether  additional  investment  in  those  activities  is 
warranted.  Also,  the  agency  risks  beginning  the  activities  before  it  has  the 
knowledge  to  complete  them  without  the  need  for  expensive  rework. 

We  are  making  recommendations  that  are  intended  to  make  the  Missile 
Defense  Agency’s  acquisition  process  more  disciplined  and  provide  better 
information  for  decision  makers  as  additional  investments  in  the  Airborne 
Laser  are  considered. 

In  commenting  on  a  draft  of  this  report,  the  Department  of  Defense 
partially  concurred  with  our  recommendations.  The  department  stated 
that  Secretary  of  Defense  direction  is  not  needed  to  implement  our 
recommendations,  the  Missile  Defense  Agency’s  acquisition  process  for 
ballistic  missile  defense  already  uses  tailored  versions  of  the  knowledge- 
based  practices  recommended  by  us,  and  the  agency  intends  to  expand  the 
use  of  knowledge-based  criteria  in  the  future.  The  Missile  Defense 
Agency’s  acquisition  process  separates  acquisition  into  three  phases — 
development,  transition,  and  production.  While  the  process  definitely 
separates  product  development  from  production,  it  clearly  does  not 
separate  technology  development  from  product  development.  Also,  it  does 
not  establish  the  knowledge-based  criteria  characteristic  of  successful 
programs  at  any  of  these  decision  points.  Because  we  have  not  seen 
evidence  in  the  Airborne  Laser’s  strategy  that  such  decision  points  and 
criteria  are  in  place,  we  have  retained  our  recommendations. 


Background 


The  effort  to  develop  the  Airborne  Laser  is  based  on  over  25  years  of 
scientific  development  in  the  Departments  of  Defense  and  Energy.  It 
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evolved  primarily  from  Airborne  Laser  laboratory  research  to  develop 
applications  for  high-energy  lasers.  This  research  culminated  in  a 
demonstration  that  showed  that  a  low-power,  short-range  laser  was 
capable  of  destroying  a  short-range  air-to-air  missile.  Although  this 
demonstration  was  considered  militarily  insignificant  because  of  the 
laser’s  low  power  and  short  range,  it  did  succeed  in  identifying 
technologies  that  were  necessary  for  the  development  of  an  operational 
Airborne  Laser  system.  The  research  showed  that  an  operational  system 
would  need  optics  that  could  compensate  for  the  atmospheric  turbulence 
that  weakens  and  scatters  a  laser  beam,  optical  devices  that  could 
withstand  the  heat  produced  by  a  high-energy  laser  without  the  added 
weight  of  water-cooling  devices,  and  a  new  chemical  laser  with  higher 
energy  levels  that  would  produce  a  stronger  laser  beam. 

In  1996,  the  Air  Force  launched  the  Airborne  Laser  program  to  develop  a 
defensive  system  that  could  destroy  enemy  missiles  from  a  distance  of 
several  hundred  kilometers.  Engineers  determined  that  if  they  were  to 
meet  this  requirement,  the  system  would  need  a  14-module^  oxygen  iodine 
laser.  They  also  determined  that  the  system  would  need  a  beam 
control/fire  control  assembly  that  could  (1)  safely  move  the  laser  beam 
through  the  aircraft,  (2)  shape  the  beam  so  that  it  was  not  scattered  or 
weakened  by  the  atmosphere,  and  (3)  hold  the  beam  on  target,  despite  the 
movement  of  the  aircraft.  In  addition,  engineers  determined  that  the 
system  would  need  a  battle  management  and  control  system  capable  of 
plarming  and  executing  an  engagement. 

The  Air  Force  planned  to  have  the  science  and  technology  community 
develop  extensive  knowledge  about  the  laser  and  beam  controFfire 
control  technologies  before  it  launched  an  Airborne  Laser  acquisition 
program.  However,  according  to  the  retired  manager  of  the  science  and 
technology  project,  the  budgets  for  technology  efforts  were  limited,  and 
the  science  and  technology  community  could  not  fund  the  technology 
maturation  effort.  The  Air  Force  knew  that  a  program  office  was  more 
likely  to  command  the  large  budget  needed  to  fully  mature  technologies, 
so  it  launched  an  acquisition  program  and  assigned  the  program  manager 
responsibility  for  both  technology  and  product  development.  The  program 
manager  planned  to  demonstrate  critical  Airborne  Laser  technologies  by 
first  building  a  six-module  version  of  the  oxygen  iodine  laser,  installing  it 


^  The  chemical  reaction  that  generates  the  laser  energy  occurs  in  the  laser  modules.  The 
amount  of  laser  energy  produced  increases  as  the  number  of  laser  modules  increases. 
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along  with  other  system  components  aboard  a  Boeing  747  aircraft  (see 
fig.  1),  and  testing  the  capability  of  this  scaled  system  in  system-level  flight 
tests.  The  tests  would  conclude  in  2003  with  an  attempt  to  shoot  down  a 
short-range  ballistic  missile  target  at  a  distance  of  100  kilometers.  If  this 
final  test  were  successful,  the  Airborne  Laser  would  have  moved  into 
product  development. 


Figure  1:  Airborne  Laser  Aboard  Boeing  747  Aircraft 


Source:  Airborne  Laser  Program  Office. 


Original  Cost  and 
Schedule  Goals  Are 
Based  on  Inadequate 
Knowledge 


The  Air  Force  launched  the  Airborne  Laser  acquisition  program  and 
identified  cost  and  schedule  goals  before  officials  had  the  knowledge  to 
make  realistic  projections.  In  1996,  when  the  program  was  launched. 
Department  of  Defense  regulation  5000.2  required,  and  still  requires  today, 
that  when  a  military  service  initiates  a  major  acquisition  program,  it  must 
establish  cost  and  schedule  goals.  However,  the  Air  Force  could  not  make 
realistic  estimates  when  it  began  the  program  because  it  had  no  way  of 
knowing  how  much  engineering  effort  would  be  needed  to  complete  the 
development  of  technology  critical  to  the  system.  Even  today,  some 
critical  technologies  that  the  system’s  design  depends  upon  remain 
immature,  making  it  very  difficult  for  analysts  to  determine  how  long  it 
will  take  and  how  much  it  will  cost  to  develop  and  produce  the  system. 


Technologies  Were  At  the  time  the  Airborne  Laser  program  was  launched,  the  laser  and  beam 

Immature  at  Program  control/fire  control  technologies  needed  to  develop  the  Airborne  Laser 

Launch  system  was  immature.  The  Department  of  Defense’s  science  and 

technology  community  was  actively  researching  and  developing  the  laser 
and  had  produced  a  weak  beam  in  a  laboratory  setting — but  this  major 
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component  had  not  reached  the  level  of  maturity  needed  to  proceed  into 
product  development.  The  technology  necessary  to  develop  the  beam 
control/fire  control  was  even  less  advanced.  Most  of  the  scientists’  work 
was  limited  to  analytical  studies,  wherein  a  few  tests  of  laboratory 
hardware  were  linked  together  to  work  somewhat  like  the  intended 
component. 

Because  technology  development  is  a  process  of  discovery,  the  Air  Force 
soon  learned  that  there  were  too  many  unknowns  regarding  the 
development  of  Airborne  Laser  technology  to  make  good  cost  and 
schedule  estimates.  As  the  technology  development  progressed, 
unanticipated  technical  challenges  affected  the  program’s  cost  and 
schedule.  Department  of  Defense  analysts  reported  that  the  Airborne 
Laser  program  experienced  cost  and  schedule  growth  because  the 
program  and  its  contractors  underestimated  the  complexity  of 
(1)  designing  laser  components,  (2)  the  system’s  engineering  analysis  and 
design  effort,  and  (3)  engineering  the  system  to  fit  on  board  the  aircraft.  As 
system  development  progressed  and  the  Air  Force  gained  a  better 
understanding  of  the  technical  complexity  of  the  system,  the  Air  Force 
increased  its  cost  and  schedule  estimates. 


Some  Critical  Technologies  The  Air  Force  has  made  some  progress  in  developing  the  Airborne  Laser’s 
Remain  Immature  critical  technologies,  but  many  remain  immature.  We  asked  the  Airborne 

Laser  program  office  to  determine  the  technologies  most  critical  to  the 
Airborne  Laser  system  and  to  use  technology  readiness  levels^  to  assess 
the  maturity  of  each.  The  officials  determined  that  if  the  Airborne  Laser  is 
to  meet  the  requirements  established  by  the  war  fighters,  then  engineers 
must  mature  technologies  in  six  areas,  all  of  which  are  needed  to 
successfully  design  the  system.  These  technologies  are 


•  devices  that  stabilize  the  laser  system  aboard  the  aircraft  so  that  the  beam 
can  be  maintained  firmly  on  the  target, 

•  optics — mirrors  and  windows — that  focus  and  control  the  laser  beam  and 
allow  it  to  pass  safely  through  the  aircraft, 

•  optical  coatings  that  enhance  the  optics’  ability  to  pass  laser  energy 
through  the  system  and  to  reflect  the  laser  energy. 


^  Technology  readiness  levels  were  developed  by  the  National  Aeronantics  and  Space 
Agency  and  are  recommended  for  use  by  the  Department  of  Defense  and  the  military 
services.  (See  appendbr  1  for  their  definition.) 
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•  hardware  that  works  in  tandem  with  computer  software  to  actively  track 
the  target  missile, 

•  devices  that  measure  atmospheric  turbulence  and  compensate  for  it  so 
that  it  does  not  scatter  or  weaken  the  laser  beam,  and 

•  safety  systems  that  automatically  shut  down  the  high  energy  laser  in  the 
event  of  an  emergency. 

At  our  request,  the  program  office  also  assessed  the  maturity  of  the 
oxygen  iodine  laser. 

As  figure  2  shows,  program  officials  assessed  the  optical  coatings  at  level 
five  and  the  safety  systems,  atmospheric  compensation,  and  target¬ 
tracking  components  at  level  six.  At  technology  readiness  level  five,  the 
technology  being  tested  is  incorporated  into  hardware  whose  form  and  fit 
are  coming  closer  to  that  needed  for  an  operational  component  and 
integrated  with  reasonable  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment.  At  level  six,  the 
technology  is  incorporated  into  a  prototype  and  tested  in  a  high-fidelity 
laboratory  environment  or  in  a  simulated  operational  environment.  The 
program  officials  identified  the  optics  and  stabilizing  devices  as  the  least 
mature — at  level  four.  At  this  level,  engineers  have  shown  that  a 
technology  is  technically  feasible  but  have  not  shown  whether  the 
technology  will  have  the  form,  fit,  or  function  required  in  the  operational 
system.  We  agreed  with  all  but  one  of  the  program  officials’  assessments 
for  these  technologies. 
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Figure  2:  Current  Airborne  Laser  Technology  Readiness  Levels 
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Acceptable  risk  for 
entry  into  product 
development 

Source:  GAO's  analysis. 

Our  one  disagreement  centered  on  the  maturity  of  the  laser  component  of 
the  system.  While  the  program  office  assessed  it  at  a  technology  readiness 
level  of  six,  we  consider  the  laser  technology  to  be  at  level  four  because 
tests  have  been  conducted  only  for  a  one-module  laser  in  a  controlled 
laboratory  environment  using  surrogate  components.  For  example,  the 
tests  used  a  stable  laser  resonator,  rather  than  the  unstable  resonator  that 


0  12  3 

Technology  readiness  levels 


Note:  Appendix  I  contains  information  describing  technology  readiness  levels. 
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will  be  used  in  system-level  flight  tests.^  We  also  found  that  during  tests  of 
the  one-module  laser,  the  resonator  was  operating  in  multimode  rather 
than  single-mode.*^  The  resonator  in  the  operational  system  will  operate  in 
single  mode.  Furthermore,  the  chemical  storage  and  delivery 
subcomponents  used  in  these  tests  were  not  representative  of  those  that 
will  be  incorporated  into  the  system’s  design.  According  to  program  office 
officials,  conducting  a  more  realistic  test  would  have  cost  time  and  money 
that  were  not  available. 

Documents  summarizing  the  tests  of  the  one-module  laser  stated  that  the 
tests  were  successful  in  reducing  the  technical  risks  associated  with  the 
one-module  system  but  that  a  new  set  of  technical  risks  linked  with 
developing  a  multimodule  system  must  still  be  addressed  during  testing  of 
the  six-module  system.  In  our  opinion,  the  program  office  will 
demonstrate  the  laser  technology  in  a  relative  environment  (technology 
readiness  level  six)  when  the  six-module  system  is  integrated  and 
successfully  tested  at  full  power  within  the  high-fidelity  laboratory 
environment  of  the  Airborne  Laser  Systems  Integration  Laboratory, 
currently  under  construction  at  Edwards  Air  Force  Base,  California. 
According  to  the  program  office,  this  type  of  demonstration  will  not  occur 
until  February  2003. 


^  A  resonator  consists  of  two  mirrors  placed  at  opposite  ends  of  a  laser  cavity.  As  the 
reaction  of  chemicals  within  the  laser  cavity  produces  photons  of  hght,  the  photons  are 
reflected  back  and  forth  between  the  two  mirrors,  which  generates  additional  photons  and 
creates  a  state  of  high  energy  within  the  cavity.  In  a  stable  resonator,  one  mirror  is  fully 
reflective  while  the  other  is  partially  reflective  and  partially  transmissive.  Energy  that 
escapes  from  the  laser  cavity  through  the  transmissive  portion  of  the  mirror  in  a  stable 
resonator  forms  a  high-energy  beam.  In  an  unstable  resonator,  both  mirrors  are  fully 
reflective,  and  one  is  much  smaller  in  diameter.  As  the  photons  are  reflected  from  the 
larger  mirror  in  the  direction  of  the  smaller  mirror,  energy  escapes  from  the  laser  cavity 
around  the  edges  of  the  smaller  mirror  and  forms  a  doughnut-shaped  beam. 

^  As  photons  are  generated  in  a  laser  resonator,  the  photons  oscillate  or  move  in  different 
ways.  A  resonator  operating  in  single-mode  suppresses  all  photons  except  those  oscillating 
at  a  certain  frequency  so  that  the  beam  produced  can  be  directed  at  one  spot  on  the  target. 
A  resonator  operating  in  multimode  does  not  suppress  any  photons,  regardless  of  their 
frequency.  While  a  multimode  resonator  directs  more  energy  toward  the  target,  all  of  that 
energy  wdll  not  be  focused  on  one  area  of  the  target. 
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New  Strategy 
Incorporates  Some 
Knowledge-Based 
Practices,  but 
Additional  Practice 
Would  Reduce 
Program  Risk 


The  Missile  Defense  Agency’s  new  strategy  for  developing  the  Airborne 
Laser  incorporates  some  of  the  knowledge-based  practices  that 
characterize  successful  programs,  but  the  agency  would  benefit  from 
adopting  another  that  would  add  greater  discipline  to  its  acquisition 
process.  The  new  strategy  allows  more  flexibility  in  setting  requirements, 
makes  time  and  facilities  available  to  mature  and  test  the  critical 
technologies,  and  collects  information  needed  to  match  the  war  fighters’ 
requirements  to  demonstrated  technology.  However,  the  agency  has  not 
established  decision  points  with  associated  knowledge-based  criteria  for 
moving  forward  from  (1)  technology  development  to  system  integration, 
(2)  system  integration  to  system  demonstration,  and  (3)  system 
demonstration  to  production.  At  each  of  these  points,  the  agency  would 
stop  to  assess  its  knowledge  and  decide  whether  investment  in  the 
program’s  next  phase  is  warranted. 


New  Strategy  Introduces  The  first  new  practice  allows  the  Missile  Defense  Agency  to  refine 
Knowledge-Based  requirements  on  the  basis  of  the  results  of  system  engineering.  The 

Practices  Department  of  Defense  ordinarily  faces  significant  hurdles  in  matching 

requirements  to  resources.  The  fundamental  problem  is  twofold.  First, 
under  the  department’s  traditional  process,  requirements  must  be  set 
before  a  program  can  be  approved  and  a  program  must  be  approved 
before  the  product  developer  conducts  systems  engineering.  Second,  the 
competition  for  funding  encourages  requirements  that  will  make  the 
desired  weapon  system  stand  out  from  others.  Consequently,  many  of  the 
department’s  product  development  programs  include  unrealistic 
requirements  set  by  the  user  before  the  product  developer  has  conducted 
the  system  engineering  necessary  to  identify  the  time,  technology,  and 
money  necessary  to  develop  a  product  capable  of  meeting  requirements. 

A  second  practice  that  is  likely  to  improve  the  Airborne  Laser’s 
development  is  making  the  time  and  facilities  available  to  mature  and  test 
critical  technologies.  To  implement  this  practice,  the  agency  increased  the 
time  available  to  test  the  six-module  laser  system  and  is  building  a  new 
test  facility.  Instead  of  following  the  Air  Force’s  plan  to  complete  system- 
level  flight  tests  of  the  six-module  system  in  the  last  quarter  of  fiscal  year 
2003,  the  agency  has  delayed  the  demonstration  to  the  first  quarter  of 
fiscal  year  2005.  This  delay  will  allow  additional  time  to  learn  from  and 
correct  problems  discovered  during  system-level  tests  that  are  scheduled 
to  begin  in  the  last  quarter  of  fiscal  year  2003  and  end  with  the  fiscal  year 
2005  demonstration.  In  addition,  the  agency  plans  to  increase  the  Airborne 
Laser’s  ground-testing  capability  by  awarding  a  contract  in  2003  for  what 
the  agency  is  calling  an  “iron  bird,”  which  is  essentially  an  aircraft  hull 
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with  installed  laser  equipment.  The  “iron  bird”  is  expected  to  allow  testing 
of  a  fully  integrated  Airborne  Laser  system  on  the  ground  so  that 
technologies  for  future  blocks  can  be  evaluated  before  being  installed  in 
an  aircraft. 

The  information  gained  from  testing  informs  the  requirements  process. 
Because  testing  allows  developers  to  gauge  the  progress  being  made  in 
translating  an  idea  into  a  weapon  system,  it  enables  the  developer  to  make 
a  more  informed  decision  as  to  whether  a  technology  is  ready  to  be 
incorporated  into  a  system’s  design.  With  this  knowledge,  the  developer 
can  determine  whether  the  technology  is  so  important  to  the  system’s 
design  that  additional  time  and  money  should  be  spent  to  mature  the 
technology  or  whether  the  system’s  initial  performance  requirements 
should  be  reduced. 

A  third  practice  that  the  agency  plans  to  adopt  is  matching  requirements  to 
available  technology.  According  to  the  Missile  Defense  Agency’s  Technical 
Director,  the  agency  defines  the  war  fighters’  requirement  as  a  system  that 
has  the  capability  to  destroy  some  threat  ballistic  missiles  during  their 
boost  phase  at  a  range  representative  of  an  operational  scenario.  The 
Technical  Director  told  us  that  the  agency  will  attain  the  knowledge  to 
determine  if  it  has  the  technology  in-hand  to  meet  this  requirement  by 
examining  each  block’s  capabilities  during  simulated  and  system-level 
flight  test  and  comparing  those  capabilities  with  data  derived  from 
intelligence  sources  on  the  likely  launch  points  and  types  of  missiles  that 
the  system  could  encounter.  Our  previous  work  with  successful 
development  programs  shows  that  once  the  technology  is  in-hand  to  meet 
the  customer’s  requirements,  the  developer  can  make  more  accurate  initial 
estimates  of  the  cost  and  time  needed  to  develop  and  produce  an 
operational  system. 
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Successful  Developers 
Recognize  Need  for 
Knowledge-Based  Decision 
Points 


Successful  developers  have  instilled  discipline  in  their  acquisition 
processes  by  requiring  that  certain  criteria  for  attaining  knowledge  are  met 
as  an  acquisition  program  moves  forward.  (See  fig.  3.)  They  recognize  that 
the  focus  and  cost  of  activities  change  over  time  and  that  less  rework  is 
required  if  all  activities  with  the  same  focus  are  completed  before 
beginning  other  activities. 


Figure  3:  The  Knowledge-Based  Process 


Knowledge  Point  1 :  Knowledge  Point  2:  Knowledge  Point  3: 

Needs  Match  Design  is  Production  can  meet  cost, 

Resources  stable  schedule,  and  quality  targets 


Source:  GAO's  analysis. 


In  successful  development  programs,  decisions  are  made  when  the 
knowledge  is  available  to  support  those  decisions.  The  first  decision  point, 
or  knowledge  point,  occurs  when  the  focus  of  a  developer’s  activities 
changes  from  technology  development  to  system  integration — the  first 
phase  of  product  development.  The  criterion  for  deciding  to  move  forward 
is  having  the  knowledge  to  match  requirements  and  available  resources 
(time,  technology,  and  funds).  The  second  knowledge  point  occurs 
between  system  integration  and  system  demonstration  when  the  developer 
has  successfully  integrated  subsystems  and  components  into  a  stable 
design  that  not  only  meets  the  customer’s  performance  requirements  but 
also  is  optimized  for  reproducibility,  maintainability,  and  reliability.  The 
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decision  criterion  used  here  is  usually  having  completed  about  90  percent 
of  the  engineering  drawings.  The  third  knowledge  point  separates  system 
demonstration  from  production.  The  decision  to  invest  in  production  is 
generally  based  on  a  determination  that  the  product  performs  as  required 
during  testing  and  that  the  manufacturing  processes  will  produce  a 
product  within  cost,  schedule,  and  quality  targets. 

The  cost  of  a  program’s  activities  increases  as  it  moves  closer  to 
production.  In  commercial  acquisitions,  product  development  is  typically 
much  more  costly  than  technology  development.  During  technology 
development,  small  teams  of  technologists  work  to  perfect  the  application 
of  scientific  knowledge  to  a  practical  problem.  As  product  development 
begins,  developers  begin  to  make  larger  investments  in  human  capital, 
bringing  on  a  large  engineering  force  to  design  and  manufacture  the 
product.  In  addition,  product  development  requires  significant  investments 
in  facilities  and  materials.  These  investments  increase  continuously  as  the 
product  approaches  the  point  of  manufacture.  In  fact,  industry  experts 
estimate  that  identifying  and  resolving  a  problem  during  product 
development  can  cost  10  times  more  than  correcting  that  problem  during 
technology  development  and  that  correcting  the  problem  during 
manufacturing  is  even  more  costly. 


Knowledge-Based  We  examined  the  Airborne  Laser’s  acquisition  strategy  and  determined 

Decisions  Missing  from  that  it  does  not  include  decision  points  at  which  officials  would  use 
Airborne  Laser  Strategy  knowledge-based  criteria  to  determine  if  the  program  is  ready  to  move 

from  technology  development  to  system  integration,  system  integration  to 
system  demonstration,  and  system  demonstration  to  production.  We  found 
that  the  agency’s  process  has  three  phases:  development,  transition,  and 
production. 

•  Development  includes  all  developmental  activities  and  system-level 
demonstrations  of  military  utility. 

•  Transition  will  involve  preparation  of  the  operational  requirements 
document  by  the  appropriate  armed  service  and  conducting  operational 
testing. 

•  Production  will  involve  producing  and  fielding  the  final  weapon  system. 

The  agency’s  strategy  also  calls  for  developing  the  Airborne  Laser 
incrementally,  rather  than  trying  to  initially  develop  a  system  with  all 
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desired  capabilities.  In  the  near  term,  the  agency  plans  to  complete  the  six- 
module  laser  system  aircraft,  now  known  as  block  2004,^  and  use  it  to 
demonstrate  critical  Airborne  Laser  technologies.  Beginning  in  March 
2003,  the  agency  intends  to  begin  developing  another  demonstration 
aircraft,  known  as  block  2008,  which  will  incorporate  new  capabilities  and 
technologies.  The  Airborne  Laser  program  manager  told  us  that  blocks 
2004  and  2008  are  primarily  test  assets  for  the  purpose  of  technology 
demonstration.  While  some  of  the  block  2008  activities  are  focused  on 
improving  subsystems  and  components,  such  as  reducing  the  weight  of 
laser  components  and  improving  optics,  other  activities  are  focused  on  the 
integration  of  these  pieces  into  a  block  2008  design.** 

The  agency  expects  to  develop  subsequent  blocks,  or  system 
configurations  to  introduce  additional  capabilities.  If  system-level  tests 
show  that  any  one  of  these  configurations  performs  at  a  level  that  merits 
fielding,  the  Air  Force  will  prepare  a  requirements  document  based  on  the 
configuration’s  demonstrated  capabilities  and  make  plans  for  operational 
testing  and  production.  This  “baseline”  capability  would  be  improved  in 
subsequent  blocks  as  more  advanced  technology  becomes  available  and  as 
the  threat  warrants. 

We  did  not  find  that  the  agency’s  strategy  includes  a  disciplined  process 
that  separates  technology  development,  system  integration,  system 
demonstration,  and  production  with  decision  points  supported  by 
knowledge-based  criteria.  Instead,  the  agency  has  put  in  place  a  decision 
point  for  moving  from  the  development  to  the  transition  phase.  According 
to  the  agency’s  strategy,  when  the  agency  determines  that  it  has  the 
technology  in-hand  to  produce  a  system  that  merits  fielding,  it  will  begin  to 
transition  the  system  over  to  the  appropriate  military  service.  Also,  at  the 
end  of  the  transition  phase,  a  system  would  enter  the  formal  Department 
of  Defense  acquisition  process  at  Milestone  C — the  point  at  which  the 


’’  The  six-module  system  is  referred  to  as  block  2004  because  testiug  will  couclude  iu 
December  2004.  Testiug  of  a  secoud  coufiguratiou,  kuowu  as  block  2008,  will  be  completed 
iu  December  2008. 

*  Oue  of  the  major  techuical  challeuges  is  accommodatiug  the  laser’s  weight.  Eugiueers 
determiued  that  the  six-module  system  would  weigh  180,000  pouuds,  but  the  origiual 
system  requiremeut  was  that  the  system  must  weigh  uo  more  thau  175,000  pouuds  with 
14  laser  modules.  Because  each  additioual  module  weighs  about  6,000  pouuds,  the  ageucy 
iuteuds  to  redesigu  some  compoueuts  to  reduce  their  mass  aud  redistribute  the  weight 
usiug  a  passeuger  versiou  of  the  Boeiug  747  as  the  block  2008  aircraft.  The  passeuger 
versiou  of  the  747  cau  accourmodate  the  crew  ou  au  upper  deck,  thereby  allowiug  the 
laser’s  weight  to  be  moved  forward  where  it  places  less  stress  ou  the  aircraft  frame. 
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decision  is  made  to  enter  low  rate  initial  production.  We  did  not  find, 
however,  an  established  set  of  decision  points  with  associated  criteria  that 
would  enable  the  agency  to  make  a  knowledge-based  decision  on  whether 
to  invest  in  system  integration  and,  subsequently,  system  demonstration 
and  production.  That  is,  even  though  the  agency  might  know  that  it  has  the 
technology  in-hand  to  develop  a  useful  military  capability,  it  has  not 
established  a  first  decision  point  where  it  would  determine  the  cost  and 
time  needed  to  move  the  program  forward  and  whether  the  program 
should  proceed  into  a  system  integration  phase  during  which  the  design 
would  be  matured  and  optimized  for  reproducibility,  maintainability,  and 
reliability.  Neither  does  the  agency’s  strategy  include  a  second  decision 
point  that  would  allow  agency  officials  to  use  the  knowledge  they  have 
attained  regarding  the  design’s  maturity  to  determine  whether  to  invest 
further  to  demonstrate  that  the  system  meets  requirements  and  that 
manufacturing  processes  are  in  place  to  repeatedly  produce  a  quality 
product.  Only  after  the  agency  successfully  moves  the  program  through  all 
of  these  decision  points  and  successfully  demonstrates  the  system’s 
capabilities  and  manufacturing  processes  would  the  agency’s  production 
decision  be  fully  knowledge  based.  Without  this  disciplined  process,  the 
agency  would  be  accepting  greater  cost  and  schedule  risks  and  is  much 
less  likely  to  realize  the  full  potential  benefits  of  its  new  approach  to 
developing  missile  defense  systems. 


Conclusion 


The  revolutionary  nature  of  missile  defense  weapon  systems  demands 
cutting-edge  technology.  Although  there  is  no  one  approach  that  ensures 
that  a  developer  can  deal  successfully  with  the  unknowns  inherent  in 
developing  a  product  from  such  technology,  the  knowledge-based  process 
has  proven  to  yield  good  results  within  cost  and  schedule  estimates.  The 
Missile  Defense  Agency  has  implemented  practices  that  are  part  of  the 
knowledge-based  approach,  and  these  practices  are  likely  to  improve  the 
agency’s  ability  to  gather  the  knowledge  it  needs  to  develop  an  Airborne 
Laser  capability  acceptable  to  the  war  fighter.  However,  the  agency  has 
the  opportunity  to  make  its  acquisition  process  more  disciplined.  By 
establishing  knowledge-based  decision  points  at  key  junctures,  the  agency 
would  be  in  a  better  position  to  decide  whether  to  move  from  one 
development  phase  to  the  next.  Also,  the  agency  would  be  better  able  to 
hold  system  developers  accountable  for  planning  all  of  the  activities 
required  to  develop  a  quality  product,  approaching  those  activities  in  a 
systematic  manner  so  that  no  important  steps  are  skipped  and  problems 
are  resolved  sooner  rather  than  later,  and  making  cost  and  schedule 
projections  when  they  have  the  knowledge  to  make  realistic  estimates. 
With  this  disciplined  process  in  place,  the  agency  is  much  more  likely  to 
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achieve  a  needed  capability  for  the  war  fighter  within  established  cost  and 
schedule  goals. 


Recommendations  for 
Executive  Action 


To  make  its  acquisition  process  more  disciplined  and  provide  better 
information  for  decision  makers  as  additional  investments  in  the  Airborne 
Laser  are  considered,  we  recommend  that  the  Secretary  of  Defense  direct 
the  Director  of  the  Missile  Defense  Agency  to  establish  decision  points 
separating  technology  development  from  system  integration,  system 
integration  from  system  demonstration,  and  system  demonstration  from 
production.  For  each  decision  point,  we  recommend  that  the  Secretary 
instruct  the  Director  to  establish  knowledge-based  criteria  and  use  those 
criteria  to  determine  where  additional  investments  should  be  made  in  the 
program. 


Agency  Comments 
and  Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  the  Department  of  Defense 
partially  concurred  with  our  recommendations  (see  appendix  11).  The 
department  stated  that  Secretary  of  Defense  direction  is  not  needed  to 
implement  our  recommendations,  the  Missile  Defense  Agency’s  acquisition 
process  for  ballistic  missile  defense  already  uses  tailored  versions  of  the 
knowledge-based  practices  recommended  by  us,  and  the  agency  intends  to 
expand  the  use  of  knowledge-based  criteria  in  the  future. 

The  Department  of  Defense  has  not  fully  implemented  the  knowledge- 
based  process  recommended  in  our  reports.  Effective  product 
development  depends  on  gaining  sufficient  knowledge  about  technology, 
design,  and  manufacturing  processes  at  key  points  in  a  system’s 
development.  At  those  points,  using  metrics-such  as  technology  readiness 
levels  to  measure  the  maturity  of  technology-that  are  commonly 
understood  allow  informed  trade-offs  to  be  made  between  resources, 
including  cost  and  time,  and  performance.  We  have  found  that  product 
development  activities,  such  as  building  engineering  prototypes  of  an 
integrated  system  and  then  demonstrating  that  the  system  can  be 
manufactured  to  acceptable  cost  and  quality  standards,  are  ineffective 
unless  the  technologies  needed  to  meet  the  product’s  intended  capabilities 
are  fully  matured  and  ready  for  system  integration.  Virtually  every  world- 
class  product  developer  we  have  spoken  with  agrees  with  this. 

The  Airborne  Laser  program  does  not  appear  to  have  established  this  type 
of  decision-making  process.  The  Missile  Defense  Agency  appears  to  have 
set  up  a  development  phase  that  combines  maturing  technologies  with 
establishing  a  stable  design.  It  does  not  include  any  visible  decision  points 
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or  standards  to  clearly  indicate  when  technology  development  is 
concluded  and  system  integration  work  to  establish  a  design  begins.  Thus, 
it  appears  to  us  that  this  acquisition  process  forces  the  agency  to  manage 
significant  risk  from  immature  technologies  simultaneously  with  trying  to 
build  a  stable  product  design  during  this  phase.  Further,  separating  system 
integration  from  system  demonstration  and  system  demonstration  from 
production  and  using  common  metrics  in  deciding  to  move  forward  will 
enhance  the  future  likelihood  that  decisions  on  the  Airborne  Laser  will  be 
cost-effective.  Such  a  process  will  also  enhance  decision-makers’  ability 
across  the  range  of  missile  defense  elements  by  facilitating  comparisons 
across  elements.  Therefore,  we  have  retained  our  recommendations. 


Scope  and 
Methodology 


To  address  our  objectives,  we  reviewed  the  contractor’s  monthly  cost 
performance  reports.  Defense  Contract  Management  Agency  analyses  of 
those  reports,  and  Defense  Acquisition  Executive  Summaries  and  Selected 
Acquisition  Reports  prepared  by  the  Airborne  Laser  program  office.  We 
also  discussed  cost  and  schedule  problems  with  Airborne  Laser  program 
officials,  Kirtland  Air  Force  Base,  New  Mexico;  and  contractor  officials  at 
the  Boeing  Company,  Seattle,  Washington;  Lockheed  Martin,  Sunnyvale, 
California;  and  TRW,  Los  Angeles,  California.  In  addition,  we  obtained  a 
technology  readiness  level  analysis  of  the  system’s  critical  technologies 
from  the  Airborne  Laser  program  office.  We  compared  this  analysis  with 
information  obtained  during  our  prior  review  to  determine  if  progress  had 
been  made  in  maturing  the  critical  technologies  to  higher  technology 
readiness  levels.  We  obtained  detailed  briefings  from  program  office 
personnel  and  Missile  Defense  Agency  officials,  Arlington,  Virginia;  and 
from  the  contractors  about  the  status  of  critical  technologies  and  the 
problems  associated  with  maturing  the  technologies  required  for  the  laser, 
the  beam  controFfire  control  system,  and  the  required  aircraft 
modifications.  We  also  obtained  detailed  briefings  from  program  office 
and  Missile  Defense  Agency  officials  regarding  the  new  Missile  Defense 
Agency  acquisition  process  and  the  implementation  of  this  process  within 
the  Airborne  Laser  program.  We  conducted  our  review  from  July  2001 
through  May  2002  in  accordance  with  generally  accepted  government 
auditing  standards. 


As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  of  this  report  to  the 
congressional  defense  committees;  the  Secretary  of  Defense;  the  Director, 
Missile  Defense  Agency,  the  Secretary  of  the  Air  Force;  and  the  Director, 
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Office  of  Management  and  Budget.  We  will  also  make  copies  available  to 
other  interested  parties  upon  request.  In  addition,  the  report  will  be 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

Please  contact  me  at  (202)  512-4841  if  you  or  your  staff  have  any  questions 
concerning  this  report.  Key  contributors  to  this  report  are  identified  in 
appendix  111. 

Sincerely  yours, 

R.  E.  Levin 

Director,  Acquisition  and 
Sourcing  Management 
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Appendix  I:  Technology  Readiness  Level 
Assessment  Matrix 


Technology  readiness  level 

Description 

1 .  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be  translated  into 
applied  research  and  development.  Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be 
invented.  The  application  is  speculative,  and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumption.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  initiated.  This  includes  analytical  studies  and 
laboratory  studies  to  physically  validate  analytical  predictions  of  separate  elements  of  the 
technology.  Examples  include  components  that  are  not  yet  integrated  or  representative 

4.  Component  and/or  breadboard. 
Validation  in  laboratory  environment. 

Basic  technological  components  are  integrated  to  estabiish  that  the  pieces  will  work 
together.  This  is  relatively  “low  fidelity”  compared  with  the  eventual  system.  Examples 
include  integration  of  “ad  hoc”  hardware  in  a  laboratory. 

5.  Component  and/or  breadboard 
validation  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment.  Examples  include  “high  fidelity” 
laboratory  integration  of  components. 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant 
environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  the  breadboard  tested 
for  technology  readiness  level  five,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology’s  demonstrated  readiness.  Examples  include  testing  a 
prototype  in  a  high-fidelity  laboratory  environment  or  in  simulated  operational 
environment. 

7.  System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from 
technology  readiness  level  six,  requiring  the  demonstration  of  an  actual  system  prototype 
in  an  operational  environment,  such  as  in  an  aircraft,  vehicle,  or  space.  Examples  include 
testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and  “flight 
qualified”  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  technology  readiness  level  represents  the  end  of  true  system 
development.  Examples  include  developmental  test  and  evaluation  of  the  system  in  its 
intended  weapon  system  to  determine  if  it  meets  design  specifications. 

9.  Actual  system  “flight  proven”  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions,  such  as 
those  encountered  in  operational  test  and  evaluation.  In  almost  all  cases,  this  is  the  end 
of  the  last  “bug  fixing”  aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3000 


ACQUISITION, 
TECHNOLOGY 
AND  LOGISTICS 


ml  ?m 


Mr.  Robert  E.  Levin 

Director,  Acquisition  and  Sourcing  Management 
U.  S.  General  Accounting  Office 
441  G.  Street,  N.W. 

Washington,  DC  20548 


Dear  Mr  Levin: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report 
“MISSILE  DEFENSE:  Knowledge-based  Decision  Making  Needed  to  Reduce  Risks  in 
Developing  Airborne  Laser,”  dated  June  25,  2002  (GAO  Code  120079/GAO-02-631). 

The  DoD  has  reviewed  the  draft  report  and  partially  concurs  with  the 
recommendations.  Specific  comments  for  each  recommendation  are  enclosed.  We  have 
recommended  some  factual  corrections.  We  also  provided  some  administrative 
comments  under  separate  cover.  My  action  officer  for  this  effort  is  Major  Mark 
Arbogast,  (703)  695-7328,  mark.arbogast@osd.mil. 

We  appreciate  the  opportunity  to  comment  on  the  draft  report. 

^Sincerely, 


Glenn  F.  Lamartin 
Director 

Strategic  and  Tactical  Systems 


Attachment 
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GAO  DRAFT  REPORT  -  DATED  JUNE  25,  2002 
GAO  CODE  120079/GAO-02-631 

“MISSILE  DEFENSE:  KNOWLEDGE-BASED  DECISION 
MAKING  NEEDED  TO  REDUCE  RISKS  IN  DEVELOPING 
AIRBORNE  LASER” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  I:  The  GAO  recommended  that  the  Secretary  of  Defense 
direct  the  Director  of  the  Missile  Defense  Agency  to: 

Establish  decision  points  separating  technology  development  from  system  integration; 
system  integration  from  system  demonstration;  and  system  demonstration  from 
production,  (p.  16/GAO  Draft  Report) 


POD  RESPONSE:  Partially  concur.  Secretary  of  Defense  direction  is  not  needed  to 
implement  the  recommendation.  The  Department  of  Defense’s  acquisition  process  for 
ballistic  missile  defense  consists  of  three  phases:  development,  transition,  and 
procurement  and  operations.  That  process  aheady  uses  the  knowledge-hased  practices 
recommended  by  the  GAO,  tailoring  them  as  appropriate  to  meet  the  objectives  of  the 
Ballistic  Missile  Defense  System  (BMDS)  program.  The  Department  plans  to 
demonstrate  system  performance  in  a  realistic  test  environment  before  the  transition 
phase  and  the  subsequent  procurement  and  operations  phase. 

RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense 
direct  the  Director  of  the  Missile  Defense  Agency  (MDA)  to: 

Establish  knowledge-based  criteria,  for  each  of  the  decision  points  mentioned  in  the 
initial  recommendation,  and  use  that  criteria  to  determine  where  additional  investments 
should  be  made  in  the  program,  (p.  16/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  Secretary  of  Defense  direction  is  not  needed  to 
implement  the  recommendation.  MDA  uses  an  event-hased  approach  to  development, 
where  a  progreim  moves  forward  based  on  its  progress  as  demonstrated  by  events,  not  by 
calendar  date.  MDA  intends  to  expand  the  use  of  knowledge-based  criteria  as  the  basis 
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for  these  key  events,  including  technology  development,  system  integration,  system 
demonstration,  and  production;  tailoring  the  criteria,  consistent  with  a  capability-based 
spiral  development  approach  directed  by  the  Secretary  of  Defense  for  the  BMDS. 
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GAO  Contact  512-4841 
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GAO’S  Mission 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accountability  of  the  federal  government  for  the 
American  people.  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  pohcy,  and  funding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  the  Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full- 
text  files  of  current  reports  and  testimony  and  an  expanding  archive  of  older 
products.  The  Web  site  features  a  search  engine  to  help  you  locate  documents 
using  key  words  and  phrases.  You  can  print  these  documents  in  their  entirety, 
including  charts  and  other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail 
this  list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  “Subscribe  to  daily 
E-mail  alert  for  newly  released  products”  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Public  Affairs 

Jeff  Nelhgan,  managing  director,  NelhganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G  Street  NW,  Room  7149 

Washington,  D.C.  20548 
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